System and method for extracting multiple feeds from a rolling-shutter sensor

ABSTRACT

An imaging system and method includes a rolling shutter sensor that captures a plurality of images of a scene, a time-varying illumination source that illuminates the scene, and a processor that receives the plurality of images from the rolling shutter sensor and separates the plurality of images into a plurality of feeds. Each of the plurality of images is captured as a series of lines. The rolling shutter sensor and the time-varying illumination source are operated synchronously to cause a plurality of on-cadence lines of the rolling shutter sensor to receive more illumination from the time-varying illumination source than a plurality of off-cadence lines of the rolling shutter sensor. Each of the plurality of feeds has a different contribution of the time-varying illumination source to an overall illumination of the scene.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to and benefit of the filing date ofU.S. Provisional Patent Application No. 62/455,137, entitled “System andMethod for Extracting Multiple Feeds from a Rolling-Shutter Sensor,”filed Feb. 6, 2017, which is hereby incorporated by reference in itsentirety.

TECHNICAL FIELD

The present disclosure is directed to imaging systems for conducting animage-guided procedure and more particularly to an imaging system forextracting multiple image or video feeds from a rolling-shutter sensor.

BACKGROUND

Medical robotic systems such as teleoperational systems used inperforming minimally invasive surgical procedures offer many benefitsover traditional open surgery techniques, including less pain, shorterhospital stays, quicker return to normal activities, minimal scarring,reduced recovery time, and less injury to tissue. Consequently, demandfor such medical teleoperational systems is strong and growing.

Examples of medical teleoperational systems include the da Vinci®Surgical System and the da Vinci® S™ Surgical System from IntuitiveSurgical, Inc., of Sunnyvale, Calif. Each of these systems includes asurgeon's console, a patient-side cart, a high performancethree-dimensional (“3-D”) vision system, and Intuitive Surgical'sproprietary EndoWrist® articulating instruments, which are modeled afterthe human wrist. When added to the motions of manipulators holding thesurgical instruments, these articulating instruments allow at least sixdegrees of freedom of motion to their end effectors, which is comparableto or even greater than the natural motions of open surgery.

During the performance of a medical procedure, images or videos of asurgical site may be captured under a variety of lighting conditions.For example, a surgical site may be viewed under normal or whitelighting for general-purpose imaging, fluorescence lighting forfluorescence imaging, and structured lighting for optical ranging ordepth estimation. Rolling-shutter sensors are one common type of imagesensor used to capture images or videos of a surgical site.

Accordingly, it would be advantageous to provide an imaging system thatsupports extracting multiple feeds from a rolling-shutter sensor, wherethe multiple feeds correspond to different lighting conditions.

SUMMARY

The embodiments of the invention are best summarized by the claims thatfollow the description.

In some embodiments, an imaging system may include a rolling shuttersensor that captures a plurality of images of a scene, a time-varyingillumination source that illuminates the scene, and a processor thatreceives the plurality of images from the rolling shutter sensor andseparates the plurality of images into a plurality of feeds. Each of theplurality of images is captured as a series of lines. The rollingshutter sensor and the time-varying illumination source are operatedsynchronously to cause a plurality of on-cadence lines of the rollingshutter sensor to receive more illumination from the time-varyingillumination source than a plurality of off-cadence lines of the rollingshutter sensor. Each of the plurality of feeds has a differentcontribution of the time-varying illumination source to an overallillumination of the scene.

In some embodiments, a method may include capturing a plurality ofimages of a scene using a rolling shutter sensor, illuminating the sceneusing a time-varying illumination source, synchronizing the rollingshutter sensor and the time-varying illumination source to cause a firstsubset of lines of the rolling shutter sensor to be on-cadence with thetime-varying illumination source and a second subset of lines to beoff-cadence, and separating the plurality of images into a plurality offeeds. Each of the plurality of feeds has a different contribution ofthe time-varying illumination source to an overall illumination of thescene.

In some embodiments, method may include receiving image data from arolling shutter sensor that is operated synchronously with atime-varying illumination source, determining on-cadence lines andoff-cadence lines from the received image data, determining acontribution of the time-varying illumination source to an overallillumination in the image data, and generating a plurality of feeds withdifferent contributions of the time-varying illumination source to theoverall illumination.

It is to be understood that both the foregoing general description andthe following detailed description are exemplary and explanatory innature and are intended to provide an understanding of the presentdisclosure without limiting the scope of the present disclosure. In thatregard, additional aspects, features, and advantages of the presentdisclosure will be apparent to one skilled in the art from the followingdetailed description.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a simplified diagram of an operating room employing a surgicalsystem with a bundled unit of medical devices according to someembodiments.

FIG. 2 is a simplified diagram of a teleoperational arm assembly holdinga bundled unit of medical devices according to some embodiments.

FIG. 3 is a simplified diagram of a distal end of a bundled unit ofmedical devices according to some embodiments.

FIG. 4 is a simplified diagram of an imaging system for extractingmultiple image or video feeds from a rolling shutter sensor according tosome embodiments.

FIGS. 5A-D are a sequence of simplified diagrams that illustrate theoperation of an imaging system to extract multiple image or video feedsfrom a rolling shutter sensor according to some embodiments.

FIG. 6 is a simplified diagram of a method for extracting multiple imageor video feeds from a rolling shutter sensor according to someembodiments.

FIG. 7 is a simplified diagram of a method for separating image datainto a plurality of image or video feeds according to some embodiments.

DETAILED DESCRIPTION

FIG. 1 illustrates, as an example, a top view of an operating room inwhich a surgical system 100 is being utilized by a Surgeon 20 forperforming a medical procedure on a Patient 40 who is lying down on anoperating table 50. One or more Assistants 30 may be positioned near thePatient 40 to assist in the procedure while the Surgeon 20 performs theprocedure teleoperatively by manipulating control devices 108, 109 on asurgeon console 10.

The surgical system 100 includes at least one arm 200 and any number ofadditional, optional arms 128 and/or 129. Thus, surgical system 100 maybe a multi-arm system and/or a single arm system, such as a single portsystem. One or more of arms 128, 129, and 200 may be mounted on apatient side cart 120. While system 100 is depicted as including aplurality of arms 128, 129, and 200 mounted to a common patient sidecart 120 for exemplary purposes, one or more arms may additionally oralternately be provided on separate carts.

In some embodiments, arms 128, 129, and/or 200 may each support a singlemedical device and/or a plurality of medical devices, such as a bundledunit 300 of medical devices. In the present example, the bundled unit300 is inserted through a single entry port 150 into the Patient 40.Although the entry port 150 is a minimally invasive incision in thepresent example, in the performance of other medical procedures, it mayinstead be a natural body orifice. The bundled unit 300 is held andmanipulated by the arm 200. Only the arm 200 is used in the presentexample. Arms 128 and 129 are swung out of the way during theperformance of the present medical procedure, because they are not beingused.

The console 10 includes a monitor 104 for displaying an image (e.g., a2-D or 3-D image) of a surgical site to the Surgeon 20, left and rightmanipulatable control devices 108, 109, a foot pedal 105, and aprocessor 102. The control devices 108, 109 may include any one or moreof a variety of input devices such as joysticks, gloves, trigger-guns,hand-operated controllers, or the like. The processor 102 may be adedicated computer integrated into the console 10 or positioned next ornear to it, or it may comprise a number of processing or controllercomponents that are distributed in a distributed processing fashionthroughout the system 100.

The console 10 is usually located in the same room as the Patient sothat the Surgeon may directly monitor the procedure, is physicallyavailable if necessary, and is able to speak to the Assistant(s)directly rather than over the telephone or other communication medium.However, it will be understood that the Surgeon can also be located in adifferent room, a completely different building, or other remotelocation from the Patient allowing for remote surgical procedures.

As shown in FIG. 3 , the bundled unit 300 may include two surgicalinstruments or tools 338, 339 and an image capturing device 340. Each ofthe surgical tools 338, 339 is associated with one of the controldevices 108, 109. The Surgeon performs a medical procedure bymanipulating the control devices 108, 109 so that the processor 102causes corresponding movement of their respectively associated surgicaltools 338, 339, while the Surgeon views the surgical site in 3-D on theconsole monitor 104 as it is captured by the image capturing device 140.

Control devices 108, 109 may be provided with at least the same degreesof freedom as their associated tools 338, 339 to provide the Surgeonwith telepresence, or the perception that the control devices 108, 109are integral with the tools 338, 339 so that the Surgeon has a strongsense of directly controlling the tools 338, 339.

The monitor 104 may be positioned near the Surgeon's hands so that itwill display a projected image that is oriented so that the Surgeonfeels that he or she is actually looking directly down onto theoperating site. To that end, images of the tools 338, 339 preferablyappear to be located substantially where the Surgeon's hands arelocated.

In addition, the real-time image may be projected into a perspectiveimage such that the Surgeon can manipulate the end effectors 322, 332 ofthe tools 338, 339 through their corresponding control devices 108, 109as if viewing the workspace in substantially true presence. By truepresence, it is meant that the presentation of an image is a trueperspective image simulating the viewpoint of an operator that isphysically manipulating the tools 338, 339. Thus, the processor 102transforms the coordinates of the tools 338, 339 to a perceived positionso that the perspective image is the image that one would see if theimage capturing device 140 was located directly behind the tools 338,339.

The processor 102 performs various functions in the system 100. Oneimportant function that it performs is to translate and transfer themechanical motion of control devices 108, 109 to arm 200 through controlsignals over bus 110 so that the Surgeon can effectively manipulate thetools 338, 339.

Although described as a processor, it is to be appreciated that theprocessor 102 may be implemented in practice by any combination ofhardware, software and firmware. Also, its functions as described hereinmay be performed by one unit or divided up among different components,each of which may be implemented in turn by any combination of hardware,software and firmware. Further, although being shown as part of or beingphysically adjacent to the console 10, the processor 102 may alsocomprise a number of subunits distributed throughout the system such asin printed circuit boards installed in the patient side cart 120 and/orthe arms 128, 129, 200, as well as, or alternatively to, the console 10.

For additional details on the construction and operation of variousaspects of a surgical system such as described herein, see, e.g.,commonly owned U.S. Pat. No. 6,493,608 “Aspects of a Control System of aMinimally Invasive Surgical Apparatus,” and commonly owned U.S. Pat. No.6,671,581 “Camera Referenced Control in a Minimally Invasive SurgicalApparatus,” which are incorporated herein by reference.

FIG. 2 illustrates, as an example, a simplified side view (notnecessarily in proportion or complete) of the arm 200 which is holdingthe bundled unit 300 of medical devices. A tool guide 270 is insertedthrough the minimally invasive incision 150 in the Patient and iscoupled to arm 200 by a guide holder 240. The bundled unit 300 may thenbe inserted into the Patient through the tool guide 270. The arm 200 ismechanically supported by a base 201 of the patient side cart 120.

Links 202, 203 are coupled together and to the base 201 throughhorizontal setup joints 204, 205. The setup joints 204, 205 in thisexample are passive joints that allow manual positioning of the arm 200when their brakes are released. For example, setup joint 204 allows link202 to be manually rotated about axis 206, and setup joint 205 allowslink 203 to be manually rotated about axis 207.

Although only two links and two setup joints are shown in this example,more or fewer of each may be used as appropriate in this and other armsin conjunction with the present invention. For example, although setupjoints 204, 205 are useful for horizontal positioning of the arm 200,additional setup joints may be included and useful for limited verticaland angular positioning of the arm 200. For major vertical positioningof the arm 200, however, the arm 200 may also be slidably moved alongthe vertical axis of the base 201 and locked in position.

The arm 200 also includes two active joints and a number of gears drivenby motors. A yaw joint 210 allows arm section 230 to rotate around anaxis 261, and a pitch joint 220 allows arm section 230 to rotate aboutan axis perpendicular to that of axis 261 and orthogonal to the plane ofthe drawing. An interface 302 comprises mating parts on the carriage 245and the proximal end of the bundled unit 300 such as motor driven gearsthat actuate movement of the surgical tools 338, 339 and image capturingunit 340 through conventional joints, cable and pulley systems.

The arm section 230 is configured so that sections 231, 232 are alwaysparallel to each other as the pitch joint 220 is rotated by its motor.As a consequence, the bundled unit 300 may be controllably moved bydriving the yaw and pitch motors so as to pivot about the pivot point262, which is generally located through manual positioning of the setupjoints 204, 205 so as to be at the point of entry into the Patient. Inaddition, the bundled unit 300 is coupled to a carriage 245 on the armsection 230 which in turn is coupled to a linear drive mechanism toextend or retract the bundled unit 300 along its insertion axis 263.

Although each of the yaw joint 210, pitch joint 220 and motor drivengears in the carriage 245 is controlled by an individual joint or gearcontroller, the controllers may be controlled by a common master/slavecontrol system so that the medical devices of the bundled unit 300 maybe controlled through user (e.g., Surgeon or operator) manipulation ofits associated control device.

FIG. 3 illustrates, as an example, a perspective view of a distal end ofthe bundled unit 300. The bundled unit 300 includes removable surgicaltools 338, 339 for performing a medical procedure and a removable imagecapturing unit 340 for viewing the procedure at a surgical site within apatient. Each of the tools 338, 339 and image capturing unit 340 extendsthrough a separate lumen formed in an inner core of the bundled unit300. Replacement of one or both of the surgical tools 338, 339 during orin preparation for performing a medical procedure may then beaccomplished by the Assistant removing the tool that is no longer neededfrom its lumen and replacing it with a substitute tool 131 from a tray60 by inserting the substitute tool 131 in the vacated lumen.Alternatively, if unused lumens are available, an additional tool may beinserted through one of those available lumens without removing anyother tools already in place.

The image capturing device 340 preferably includes a stereoscopic pairof cameras 342, 343 (and/or a single binocular camera) forthree-dimensional imaging of the surgical site and an illuminatingdevice 344 such as a light emitting diode (LED) or a fiber optics bundlecarrying light from an external source, to enhance visibility of objectsin the captured images. Auxiliary image capturing units, such as anultrasound probe, may also be provided in available lumens of thebundled unit 300 for “seeing” into anatomic structures for surgical ordiagnostic purposes.

In some embodiments, an overtube 310 is also included in the bundledunit 300 for protecting its inner core and the medical devices (i.e.,surgical tools and image capturing units) inserted therethrough. Theovertube 310 may be rigid. Alternatively, it may be formed of flexiblematerial or comprise actively and/or passively bendable sections so thatthe bundled unit 300 may conform to the shapes of body lumens as itmoves therethrough to a surgical site within a patient.

The surgical tools 338, 339 each have a controllably extendable,rotatable, and bendable arm to which their respective end effectors 322,332 are coupled to by wrist mechanisms 323, 337. For example, the arm ofthe surgical tool 339 comprises three links 331, 333, 335 coupled bydistal joints 334, 336. The proximal link 335 is controllably extendableand retractable along an insertion axis 352 (which is preferablyparallel to the insertion axis 263 of the single-port device 300), andis controllably rotatable (as shown by rotation angle 353) about theinsertion axis 352. The middle link 333, on the other hand, iscontrollably bendable by distal joint 336 relative to the link 335 (asshown by bend angle 351), and the distal link 331 is coupled to thelinks 333, 335 and bendable by distal joint 334 so that its bend angle354 is in an opposite direction as that of the link 333 andconsequently, keeps links 331, 335 in parallel alignment.

The arm of the surgical tool 338 is similarly constructed as that of thesurgical tool 339. Additional details for one example of the wristmechanisms 323, 337 are provided in commonly owned U.S. Pat. No.6,817,974 “Surgical Tool Having Positively Positionable Tendon-ActuatedMulti-Disk Wrist Joint,” which is incorporated herein by this reference.

The image capturing device 340 also has a controllably extendable,rotatable, and bendable arm 345 that facilitates at leastinsertion/retraction of the image capturing unit 340 along its insertionaxis (which may be parallel to the insertion axis 263 of the single-portdevice 300) and pitch motion in order to achieve a sufficient elevationof the image capturing device 340 “above” the surgical tools 338, 339 soas to properly view them during a surgical procedure. Additional degreesof freedom, such as roll angular movement of the image capturing device340 about its insertion axis, may also be provided in order tofacilitate additional positioning and orientation capabilities for theimage capturing device 340. For enhanced maneuverability, the imagecapturing arm 345 may also be bendable such as the controllablybendable, rotatable, and extendable arms of the surgical tools 338, 339.

FIG. 4 is a simplified diagram of an imaging system 400 for extractingmultiple image feeds, such a video feeds, from a rolling shutter sensoraccording to some embodiments. According to some embodiments consistentwith FIGS. 1-3 , imaging system 400 may be incorporated into a surgicalsystem, such as surgical system 100. However, in some examples, imagingsystem 400 may be used for applications independent of the medicalteleoperational system, such as standalone imaging applications and/ornon-surgical imaging applications.

Imaging system 400 includes a rolling shutter sensor 410 that capturesimages or video of a scene 415. Rolling shutter sensor 410 operates byscanning across scene 415 line by line (e.g., in a horizontal orvertical sequence) to form an image or frame. Thus, a single frame iscaptured piecemeal, over a duration of time. By way of distinction, aglobal shutter sensor captures an entire image at the same point intime. In some examples, rolling shutter sensor 410 may be acomplementary metal-oxide-semiconductor (CMOS) imager. CMOS imagers areavailable in very small sizes. For example, some CMOS imagers are smallenough to be used in chip-on-tip endoscopes, where the image sensor islocated at a distal end of an endoscope and inserted into the patientbody. Accordingly, rolling shutter sensor 410 may be integrated into achip-on-tip endoscope for use in surgical procedures. Consistent withsuch embodiments, rolling shutter sensor 410 may be disposed in anendoscope shaft, such as an 8.8 mm endoscope shaft.

Rolling shutter sensor 410 is associated with a frame rate and a linerate. The line rate is equal to the frame rate times the number of lineson the sensor. For example, rolling shutter sensor 410 may have a framerate of 60 Hz and a sensor with 1024 vertical lines, in which case theline rate is 61.44 kHz when scanning vertically.

Imaging system 400 also includes focusing optics 412. Focusing optics412 project one or more images of scene 415 onto rolling shutter sensor410. Focusing optics 412 may include one or more lenses, shutters,apertures, reflectors, prisms, filters, and/or the like. In someexamples, focusing optics 412 may project a single image of scene 415onto rolling shutter sensor 410 for two-dimensional imaging. In someexamples, focusing optics 412 may include binocular focusing optics toproject a pair of images onto different sections of rolling shuttersensor 410. Binocular focusing optics can be used to achievethree-dimensional and/or depth imaging of scene 415. In some examples,focusing optics 412 may be mounted to the end of an endoscope shaft forchip-on-tip endoscope applications.

A time-varying illumination source 420, such as a pulsed illuminationsource, illuminates scene 415 while rolling shutter sensor 410 capturesimages of scene 415. Time-varying illumination source 420 may be asource of virtually any type of electromagnetic radiation, includingnarrowband, broadband, coherent, non-coherent, isotropic, anisotropic,visible, infrared, and/or ultraviolet radiation and may be a pointsource and/or a distributed/diffuse source. According to someembodiments, time-varying illumination source 420 may include a solidstate source of narrowband illumination, such as a solid state laserand/or a light emitting diode (LED). One or more output characteristicsof time-varying illumination source 420, such as the intensity, spectralcharacteristics, spatial distribution, and/or the like, is modulated asa function of time. In some examples, the output characteristics may bemodulated periodically as a function of time. For example, time-varyingillumination source 420 may output a stream of equally spaced pulses ofillumination and/or illumination with a cyclically varying intensity. Insome examples, the time-varying illumination source 420 may be modulatedusing a time-varying electrical power supply that provides atime-varying current and/or voltage to control the output illumination,such as when time-varying illumination source 420 includes a solid statesource. Alternately or additionally, time-varying illumination source420 may be modulated using mode locking techniques, such as whentime-varying illumination source 420 includes a mode-locked laser.

The modulation frequency of time-varying illumination source 420 (e.g.,the pulse rate) may be significantly higher than the frame rate ofrolling shutter sensor 410. In some embodiments, the modulationfrequency may approach or exceed the line rate of rolling shutter sensor410. For instance, where the frame rate is 60 Hz and the line rate is 60kHz, a pulse rate of time-varying illumination source 420 may be 6 kHzor greater.

In some embodiments, time-varying illumination source 420 may be afluorescence illumination source. Consistent with such embodiments, oneor more objects or features in scene 415, such as an object 416, may belabeled using a fluorescent agent that absorbs and re-emits illuminationfrom time-varying illumination source 420, such as a fluorescent dyeand/or a fluorescent protein. Consequently, the fluorescenceillumination source operates as an excitation source for the fluorescentagent. Other objects or features, such as an object 417, may not belabeled using the fluorescent agent.

The excitation and/or emission spectrum of the fluorescent agent may beseparated from the visible light range and/or may overlap with thevisible light range. An example of a fluorescent agent that with anexcitation and emission spectrum that is separated from the visiblelight range is indocyanine green (ICG). Examples of fluorescent agentswith excitation and/or emission spectra that overlap with the visiblelight range include fluorescein and a visible fluorescent proteins(e.g., green (GFP), yellow (YFP), blue (BFP), and/or cyan (CFP)fluorescent protein). The wavelength of the fluorescence illuminationsource is selected to match the excitation spectrum of the fluorescentagent, such as a wavelength in the blue or ultraviolet range when usingfluorescein or GFP. For example, the fluorescence illumination sourcemay be a narrow-band and/or a single-band illumination source, such as alaser.

In some embodiments, time-varying illumination source 420 may be astructured illumination source. That is, the output illumination mayvary spatially. For example, the output illumination may be a stripepattern, dot pattern, grid pattern, and/or the like. The illuminationpattern output by the structured illumination source may be used forthree dimensional positioning and/or depth estimation (e.g., opticalranging) based on deviations from the illumination pattern in thecaptured images. In some embodiments, the wavelength of the structuredillumination source may be in the near-IR range.

A continuous illumination source 430 is optionally included to providecontinuous illumination to scene 415. Unlike time-varying illuminationsource 420, the output illumination of continuous illumination source430 is substantially constant over time (i.e., varying at a rate that isslower than the frame rate of rolling shutter sensor 410). In someexamples, continuous illumination source 430 may output broadband and/orwhite light. Continuous illumination source 430 causes an image of scene410 to be projected onto rolling shutter sensor 410 even when the outputintensity of time-varying illumination source 420 is low or zero.According to some embodiments, however, continuous illumination source430 may be omitted from imaging system 400, such as when there issufficient ambient illumination to perform imaging without additionallighting.

Rolling shutter sensor 410 and time-varying illumination source 420 areoperated synchronously such that a first predetermined subset of linesin rolling shutter sensor 410 (“on-cadence” lines) receives moreillumination from time-varying illumination source 420 than a secondpredetermined subset of lines (“off-cadence” lines). In some examples,the off-cadence lines may receive no illumination from time-varyingillumination source 420.

For example, time-varying illumination source 420 may output a stream ofpulses at a pulse rate that is synchronized with the line rate ofrolling shutter sensor 410. For instance, when time-varying illuminationsource 420 includes a mode-locked laser, the repetition rate of themode-locked laser may be synchronized with the line rate of rollingshutter sensor 410. Due to the synchronous operation, a first(on-cadence) line of rolling shutter sensor 410 may receive more pulsesof illumination from time-varying illumination source 420 than asubsequent second (off-cadence) line. In some examples, the pulse ratemay further be synchronized with the frame rate of rolling shuttersensor 410. Accordingly, a particular line of rolling shutter sensor 410may switch between on-cadence and off-cadence in alternating frames. Thesynchronous operation of rolling shutter sensor 410 and time-varyingillumination source 420 is explained in greater detail in FIG. 5 below.

A synchronization module 440 may be used to achieve synchronizationbetween rolling shutter sensor 410 and time-varying illumination source420. Synchronization module 440 may include one or more clocks and/oroscillators. According to some embodiments, synchronization module 440may include a clock that generates and sends synchronized timing signalsto each of rolling shutter sensor 410 and time-varying illuminationsource 420. Synchronization module 440 may further include one or moredelay generators and/or frequency dividers to adjust the phase and/orfrequency relationships, respectively, between the synchronized timingsignals. The delay generators and/or frequency dividers may beprogrammable and/or fixed. According to some embodiments, rollingshutter sensor 410 and/or time-varying illumination source 420 mayinclude independent clocks that are synchronized based on a one-timeinitialization process at startup and/or an ongoing synchronizationprocess that may correct for clock drift.

A processor 450 receives image data from rolling shutter sensor 410 andseparates the image data into a plurality of images or video feeds. Eachof the plurality of video feeds has a different contribution oftime-varying illumination source 420 to the overall illumination ofscene 415 (e.g. a total illumination that may also include the whitelight from the continuous illumination source and/or an ambient light).For example, a first video feed may be a general-purpose video feed withlittle or no contribution from time-varying illumination source 420. Thegeneral-purpose video feed may capture scene 415 illuminated bycontinuous white light and/or ambient illumination. By contrast, asecond video feed may be a special-purpose video feed capturing thescene 415 illuminated by time-varying illumination source 420 inaddition to the continuous white light and/or ambient light. Forexample, the second video feed may capture fluorescence images of scene415 (i.e., when time-varying illumination source 420 is a fluorescenceillumination source) and/or structured images of scene 415 (i.e., whentime-varying illumination source 420 is a structured illuminationsource).

According to some embodiments, processor 450 may separate the receivedimage data into a plurality of video feeds. For example, the receivedimage data may be separated based on an illumination difference betweenthe on-cadence and off-cadence lines in the received image data. Forexample, a particular line in the image data may switch betweenon-cadence and off-cadence in alternating frames. By comparing theexposure level of the off-cadence lines to the on-cadence lines,processor 450 may determine the relative contribution of time-varyingillumination source 420 to the overall illumination of the correspondingregion of scene 415. In various embodiments, the comparison of exposurelevel may be made be a pixel-level comparison. Once the relativecontribution of time-varying illumination source 420 is known for eachline of the image data, processor 450 may separate the image data intotwo or more video feeds based on the different contributions fromtime-varying illumination source 420. For example, processor 450 mayidentify, based on the difference in illumination exposure, which linesare exposed to time-varying illumination source 420. Then, based on thisdifferential, the processor 450 may remove (e.g. filter out) thecontribution of time-varying illumination source 420 from the image datafor each image frame to generate the general-purpose video feed, and theprocessor 450 may isolate the contribution of time-varying illuminationsource 420 to the image data for each image frame to generate thespecial-purpose video feed.

Processor 450 may separate the received image data into a plurality ofvideo feeds as described above even when the spectrum of time-varyingillumination source 420 (and/or the emission spectra of fluorescentagents activated by time-varying illumination source 420) overlaps withthe spectrum of other sources of illumination to scene 415. For example,rolling shutter sensor 410, time-varying illumination source 420, andcontinuous illumination source 420 may all operate in the visible lightrange. This avoids placing constraints—which may be inconvenient,costly, and/or impractical—on the spectral characteristics oftime-varying illumination source 420, the types of fluorescent agentsthat are used for fluorescent imaging, and the filtering components, ifany, that are included in focusing optics 412. Moreover, imaging system400 may be used without spectral filters that are matched to aparticular fluorescent agent, and may therefore be compatible with avariety of fluorescent agents with different excitation and/or emissionspectra.

A display 460 is optionally included to show images or videocorresponding to the plurality of video feeds output by processor 450.In some examples, display 460 may be a component of the surgeon'sconsole, such as console 10. In some examples, display 460 mayconcurrently show two or more real-time video feeds, each of which arederived from image data captured by rolling shutter sensor 410. Forexample, the two or more real-time video feeds may include ageneral-purpose video feed (e.g., images illuminated with white and/orambient lighting) and a special purpose video feed (e.g., imagesilluminated with fluorescence illumination). In some examples, display460 may show a single video feed that combines data from the pluralityof video feeds output by processor 450. For example, the combined videofeed may show images in which the special purpose video feed is overlaidon the general-purpose video feed. In another example, the combinedvideo feed may show three-dimensional images of scene 415 based on acombination of the general-purpose video feed (which may providetwo-dimensional images) and the special-purpose video feed (which mayprovide depth information derived from structured lighting of scene415). Consistent with some embodiments, processor 450 may determinedepth information based on the structured lighting of scene 415. In someexamples, the

It is to be understood that FIG. 4 is merely an example, and variousalternatives are possible. For example, although rolling shutter sensor410 is described as scanning scene 415 line by line (i.e. a row orcolumn of pixels at a time), other pixel groupings are possible. Forexample, the groupings may be a fraction of a line, multiple lines, asingle pixel, and/or the like. Moreover, although consecutive lines in aparticular frame, and a particular line in consecutive frames, aredescribed as alternating between on-cadence and off-cadence, variousother patterns are possible. The pattern of on-cadence and off-cadencelines depends generally on the relationship between the line rate ofrolling shutter sensor 410 and the modulation frequency of time-varyingillumination source 420. For example, when the pulse rate is one thirdor one fourth of the line rate every third or fourth line, respectively,may be on-cadence. In some examples, imaging system 400 may include morethan one time-varying illumination source and more than onecorresponding special-purpose video feed. For example, the video feedsoutput by processor 450 may include a general-purpose video feed and twoor more special-purpose video feeds corresponding to fluorescenceimaging of two or more different fluorescent agents (i.e., fluorescentagents with different excitation and/or emission spectra). Similarly,the plurality of video feeds output by processor 450 may include ageneral-purpose video feed, one or more special-purpose video feedscorresponding to fluorescent imaging, and one or more special-purposevideo feeds corresponded to structured light imaging.

FIGS. 5A-D are a sequence of simplified diagrams that illustrate theoperation of an imaging system, such as imaging system 400, to extractmultiple image or video feeds from a rolling shutter sensor according tosome embodiments. As depicted in FIG. 5A, a plot 500 illustrates thesynchronous operation of a rolling shutter sensor, such as rollingshutter sensor 410, and a time-varying illumination source, such astime-varying illumination source 420. The rolling shutter sensor in thisexample has nine lines (these may be either vertical or horizontallines) numbered one through nine. Plot 500 depicts the shutter positions(i.e., the exposed lines) of the rolling shutter sensor as a function oftime during the capture of two consecutive frames.

Shaded boxes 510 represent the shutter position of the rolling shuttersensor as a function of time. In this example, the shutter has a widthof three lines. That is, at any given time, three neighboring lines ofrolling shutter sensor are exposed. The shutter position shifts downwardone line at a time at the line rate of the rolling shutter sensor. Aftereach line of the rolling shutter sensor has been exposed, the shutterposition loops back to the top line and begins to capture the nextframe. In a given frame, each line is exposed for three consecutive timeslots (a time slot corresponds to the length of time that the shutterremains at a particular position, which is the inverse of the linerate).

Arrows 520 represent a plurality of pulses of illumination output by thetime-varying illumination source. As demonstrated in plot 500, thetime-varying illumination source and the rolling shutter sensor aresynchronized. More specifically, the pulse rate of the time-varyingillumination source is half the line rate of the rolling shutter sensor.Consequently, there is one pulse of illumination in every other timeslot. As depicted in plot 500, the pulses occur near the middle of agiven time slot, but it is to be understood that the pulses may occur atany point in a given time slot. For example, it may be desirable for thepulses to occur at the beginning of a time slot due to allow forfluorescence emission to decay before the next time slot begins.According to some embodiments, rather than a stream of pulses, thetime-varying illumination source may vary cyclically over time accordingto some other function, such as a sinusoidal function. Consistent withsuch embodiments, arrows 520 may represent the peak intensity of thecyclically varying illumination source.

As depicted in plot 500, alternating lines of the rolling shutter sensorare either on-cadence or off-cadence. The on-cadence lines receive twopulses per frame, and the off-cadence lines receive one pulse per frame.For example, in the first frame, odd lines 1, 3, 5, 7, and 9 areon-cadence, and even lines 2, 4, 6, and 8 are off-cadence. In the secondframe, the cadence is reversed: even lines 2, 4, 6, and 8 are on-cadenceand odd lines 1, 3, 5, 7, and 9 are off-cadence.

In an alternative embodiment, the timing of the pulsed illumination maybe adjusted and synchronized with the rolling shutter sensor such thatthe on-cadence lines receive a single pulse per frame and theoff-cadence lines receive no pulses per frame.

FIG. 5B depicts a first frame 530 a, and FIG. 5C depicts a second frame530 b, both captured by a rolling shutter sensor during the operationillustrated in plot 500. An object 532 and an object 534 appear in eachof first and second frames 530 a-b. Object 532 and object 534 maygenerally correspond to object 416 and object 417, respectively, asdepicted in FIG. 4 . For example, object 532 may be labeled using afluorescent tag, whereas object 534 may not be labeled using thefluorescent tag. Consequently, when the time-varying illumination sourceis a fluorescent illumination source, object 532 may emit light inresponse to each pulse output by the time-varying illumination source.Meanwhile, the appearance of object 534 may be constant over time, asobject 534 does not emit light in response to the time-varyingillumination source.

Because object 532 is responsive to the pulsed light source, lines thatare on-cadence appear brighter than lines that are off-cadence,resulting in a striped appearance. In particular, odd lines 3, 5, and 7of first frame 530 a and even lines 2, 4, and 6 of second frame 530 bare on-cadence (brighter) and even lines 2, 4, and 6 of first frame 530a and odd lines 3, 5, and 7 of second frame 530 b are off-cadence(dimmer). By contrast, object 534 has the same brightness in all linesbecause it is not responsive to the pulsed light source.

In FIG. 5D, a processor 540 receives image data 550 (which includesfirst and second frames 530 a-b) and separates image data 550 into afirst video feed 560 and a second video feed 570. According to someembodiments, processor 540 may determine the contribution of thetime-varying illumination source to image data 550 based on thedifference in illumination between consecutive frames, such as first andsecond frames 530 a-b. Processor 540 then outputs first video feed 560,in which the contribution of the time-varying illumination source isremoved. That is, objects 532 and 534 are shown in first video feed 560as they appear under ambient or continuous illumination. In addition,processor 540 outputs second video feed 570, in which the contributionof the time-varying illumination source is isolated. That is, onlyobject 532 is shown in second video feed 570, as it appears undertime-varying illumination. In some examples, first video feed 560 may bea general-purpose video feed that shows objects under white light and/orambient lighting conditions, and second video feed 570 may be aspecial-purpose video feed that shows objects under, for example,fluorescence and/or structured illumination.

FIG. 6 is a simplified diagram of a method 600 for extracting multipleimage or video feeds from a rolling shutter sensor according to someembodiments. According to some embodiments consistent with FIGS. 1-5 ,method 600 may be performed during the operation of an imaging system,such as imaging system 400.

At a process 610, a plurality of images of a scene, such as scene 415,is captured using a rolling shutter sensor, such as rolling shuttersensor 410. The plurality of images are captured at a frame rate of therolling shutter sensor. Each frame or image in the plurality of imagesis captured by scanning the scene line by line (e.g., vertically orhorizontally) at a line rate of the rolling shutter sensor. As discussedpreviously with respect to FIG. 4 , the rolling shutter sensor may be aCMOS imager incorporated into a chip-on-tip endoscope.

At a process 620, the scene is illuminated using a time-varying ortime-varying illumination source, such as time-varying illuminationsource 420. According to some embodiments, the time-varying illuminationsource may be a fluorescence illumination source that causes objects orfeatures in the scene that are labeled with fluorescent tags tofluoresce. According to some embodiments, the time-varying illuminationsource may be a structured illumination source that outputs a pattern(e.g., a dot, stripe, or grid pattern) used for optical ranging and/ordepth estimation. The time-varying illumination source is operated at apulse rate that is much greater than the frame rate of the rollingshutter sensor, as discussed previously with respect to FIG. 4 .

At an optional process 630, the scene is illuminated using a continuousillumination source, such as continuous illumination source 430. Forexample, the continuous illumination source may be a white light sourcethat enhances the overall visibility of objects in the scene. In someexamples, process 630 may be omitted, such as when the scene isilluminated by ambient light.

At a process 640, the operation of the rolling shutter sensor and thetime-varying illumination source is synchronized to cause a first subsetof lines captured by the rolling shutter sensor to be on-cadence withthe time-varying illumination source and a second subset of lines to beoff-cadence. The on-cadence lines receive more illumination from thetime-varying illumination source than the off-cadence lines. In someexamples, consecutive lines in a given frame or image may alternatebetween on-cadence and off-cadence. Similarly, a given line mayalternate between on-cadence and off-cadence in consecutive frames. Forexample, as depicted in FIG. 5 , odd and even lines are on-cadence andoff-cadence, respectively, in the first frame. Meanwhile, the oppositeis true in the second frame: odd lines are off-cadence and even linesare on-cadence. The particular ratio between the amount of illuminationcaptured in the on-cadence lines versus the off-cadence lines depends onthe relationship between the line rate of the rolling shutter sensor,the modulation frequency (e.g., the pulse rate) of the time-varyingillumination source, and the shutter size. For examples, when the pulserate is half the line rate and the shutter size is three lines (i.e.,the configuration illustrated in FIG. 5 ), the on-cadence lines receivetwice as much illumination from the time-varying illumination source asthe off-cadence lines.

At a process 650, the plurality of images captured by the rollingshutter sensor is separated into a plurality of image or video feedsbased on the difference between the on-cadence and off-cadence lines.For example, when a given line alternates between on-cadence andoff-cadence in consecutive frames, a line from the first frame may becompared pixel-by-pixel to the same line in the next frame to determinethe pixel by pixel difference. In each of the plurality of video feeds,the contribution of the time-varying illumination source to the overallillumination of the scene is different. In some embodiments, thecontribution of the time-varying illumination source may be removed froma first or general-purpose video feed and the contribution of thetime-varying illumination source may be isolated in a second orspecial-purpose video feed. For example, the general-purpose video feedmay correspond to the scene as illuminated by an ambient light sourceand/or the continuous illumination source of process 630. A firstspecial-purpose video feed may correspond to fluorescence imaging thatallows for a visualization of objects or features that are labeled usingfluorescent tags. A second special-purpose video feed may correspond tostructured illumination that allows for the depth of objects or featuresto be estimated. In some example, more than one fluorescent agent may beused, in which case one or more additional special-purpose video feedsmay be separated at process 650.

At an optional process 660, one or more real-time image or video feedsare displayed to a user, such as Surgeon 20, via a display, such assurgeon console 10 and/or display 460. In some examples, thegeneral-purpose video feed and the special-purpose video feed may beconcurrently displayed to the user. For example, in endoscopyapplications, the general-purpose video feed may be used fornavigational purposes to guide the endoscope through the patientanatomy, while the special-purpose video feed may be used for clinicalpurposes such as to identify anatomical features that are labeled usingfluorescent tags. Advantageously, each of the video feeds are capturedconcurrently and displayed in real-time, which eliminates a step ofmanually switching between the general-purpose video feed and thespecial-purpose video feed. In some embodiments, a single video feed maybe displayed that combines information from the general-purpose videofeed and the special-purpose video feed so as to enhance the amount ofinformation conveyed to the operator within the single video feed. Forexample, the combined video feed may show images in which the specialpurpose video feed (e.g., fluorescent video) is overlaid on thegeneral-purpose video feed. In some examples, the single video feed maydisplay three-dimensional image data that combines two dimensionalimages from the general-purpose video feed with depth informationextracted from the special-purpose video feed.

FIG. 7 is a simplified diagram of a method 700 for separating image datainto a plurality of image or video feeds according to some embodiments.According to some embodiments consistent with FIGS. 1-6 , method 700 maybe carried out by a processor of an imaging system, such as processor450 and/or 540. In some examples, method 700 may be used to implementone or more processes of method 600, such as process 650.

At a process 710, image data is received from a rolling shutter sensor,such as rolling-shutter sensor 410, that is operated synchronously witha time-varying illumination source, such as time-varying illuminationsource 420. The image data corresponds to a plurality of frames capturedby the rolling shutter sensor. For example, the image data may becaptured using a process corresponding to processes 610-640 as describedabove.

At a process 720, on-cadence and off-cadence lines of the received imagedata are determined. The on-cadence lines are those lines that are timedto receive more illumination from the time-varying illumination sourcethan the off-cadence lines. In some examples, consecutive lines in agiven frame may alternate between on-cadence and off-cadence. In someexamples, a given line may alternate between on-cadence and off-cadencein consecutive frames. The determination of which lines are on- andoff-cadence may include receiving a timing signal from the rollingshutter sensor, the pulsed light source, and/or a synchronization moduleresponsible for synchronizing the operation of the rolling shuttersensor and the pulsed light source. Based on the timing signal,on-cadence and off-cadence lines may be identified. In some examples,on-cadence and off-cadence lines may be determined based on a prioriknowledge of the relationship between the line rate of the rollingshutter sensor and the pulse rate of the time-varying illuminationsource.

At a process 730, a contribution of the time-varying illumination sourceto the overall illumination in the image data is determined based on adifference between the on-cadence and off-cadence lines. In someembodiments, for each line of the rolling shutter sensor, apixel-by-pixel intensity difference between an on-cadence version of aline from one frame and an off-cadence version of the same line fromanother frame is determined. For example, when a given line alternatesbetween on-cadence and off-cadence in consecutive frames, the on-cadenceversion of the line and the off-cadence version of the line are takenfrom neighboring frames. The difference between versions may bedetermined by subtracting and/or dividing two version of the line. Basedon this difference, the contribution (e.g., a ratio and/or an absolutemeasure of illumination intensity) of the time-varying illuminationsource to the overall illumination may

At a process 740, a plurality of image or video feeds with differentcontributions of the time-varying illumination source are generated. Forexample, a general-purpose video feed may be generated by modifying theimage data to remove the contribution of the time-varying illuminationsource. Similarly, a special-purpose video feed may be generated byisolating the contribution of the time-varying illumination source. Forexample, for a pair of consecutive frames, the special-purpose videofeed may depict the differential intensity between the pair of frames,and the common-mode intensity may be removed.

Some examples of processors, such as processor 450 and/or 540 mayinclude non-transient, tangible, machine readable media that includeexecutable code that when run by one or more processors (e.g., processor450 and/or 540) may cause the one or more processors to perform theprocesses of methods 600 and/or 700. Some common forms of machinereadable media that may include the processes of methods 600 and/or 700are, for example, floppy disk, flexible disk, hard disk, magnetic tape,any other magnetic medium, CD-ROM, any other optical medium, punchcards, paper tape, any other physical medium with patterns of holes,RAM, PROM, EPROM, FLASH-EPROM, any other memory chip or cartridge,and/or any other medium from which a processor or computer is adapted toread.

Although illustrative embodiments have been shown and described, a widerange of modification, change and substitution is contemplated in theforegoing disclosure and in some instances, some features of theembodiments may be employed without a corresponding use of otherfeatures. One of ordinary skill in the art would recognize manyvariations, alternatives, and modifications. Thus, the scope of theinvention should be limited only by the following claims, and it isappropriate that the claims be construed broadly and in a mannerconsistent with the scope of the embodiments disclosed herein.

1-62. (canceled)
 63. An imaging system, comprising: a rolling shuttersensor that captures a plurality of images of a scene; a continuousillumination source that illuminates the scene; a time-varyingillumination source that illuminates the scene; and a processorconfigured to: synchronize the rolling shutter sensor and thetime-varying illumination source to cause a first subset of lines of therolling shutter sensor to be on-cadence with the time-varyingillumination source and a second subset of lines to be off-cadence; andseparate the plurality of images into a plurality of feeds, wherein eachof the plurality of feeds has a different contribution of thetime-varying illumination source to an overall illumination of thescene, and wherein the plurality of feeds comprise a general-purposefeed from which the contribution of the time-varying illumination sourceto the overall illumination of the scene is removed and aspecial-purpose feed in which the contribution of the time-varyingillumination source is isolated.
 64. The imaging system of claim 63,wherein the processor is configured to concurrently display theplurality of feeds to a user in real-time.
 65. The imaging system ofclaim 63, wherein the rolling shutter sensor is a complementarymetal-oxide-semiconductor (CMOS) imager.
 66. The imaging system of claim63, wherein the rolling shutter sensor is integrated into a chip-on-tipendoscope.
 67. The imaging system of claim 63, wherein the rollingshutter sensor is operated at a frame rate of 60 Hz and a line rate of61.44 kHz.
 68. The imaging system of claim 63, wherein illuminating thescene includes pulsing the time-varying illumination source to generatea stream of equally spaced pulses.
 69. The imaging system of claim 63,wherein a modulation frequency of the time-varying illumination sourceis significantly higher than a frame rate of the rolling shutter sensor.70. The imaging system of claim 63, wherein the time-varyingillumination source is a solid state illumination source.
 71. Theimaging system of claim 63, wherein the processor is further configuredto modulate the time-varying illumination source using a time-varyingelectrical power supply.
 72. The imaging system of claim 63, wherein thetime-varying illumination source is a mode-locked laser that has arepetition rate synchronized with a line rate of the rolling shuttersensor.
 73. The imaging system of claim 63, wherein the time-varyingillumination source is a source of narrowband illumination.
 74. Theimaging system of claim 63, wherein the time-varying illumination sourceis a fluorescence illumination source with a wavelength corresponding toan excitation spectrum of a fluorescent agent in the scene.
 75. Theimaging system of claim 63, wherein the time-varying illumination sourceis a structured illumination source.
 76. The imaging system of claim 75,wherein the processor is further configured to perform depth estimationbased on an illumination pattern output by the structured illuminationsource.
 77. The imaging system of claim 63, wherein the processor isfurther configured to generate and transmit synchronized timing signalsto each of the rolling shutter sensor and the time-varying illuminationsource.
 78. An imaging system comprising: a time-varying illuminationsource that illuminates a scene; a rolling shutter sensor that capturesimage data, the rolling shutter sensor operated synchronously with thetime-varying illumination source; a continuous illumination source thatilluminates the scene; and a processor configured to: determineon-cadence lines and off-cadence lines from the captured image data;determine a contribution of the time-varying illumination source to anoverall illumination in the image data; and generate a plurality offeeds with different contributions of the time-varying illuminationsource to the overall illumination, wherein the plurality of feedscomprise a general-purpose feed from which the contribution of thetime-varying illumination source to the overall illumination of a sceneis removed and a special-purpose feed in which the contribution of thetime-varying illumination source is isolated.
 79. The imaging system ofclaim 78, wherein the processor is further configured to concurrentlydisplay the plurality of feeds to a user in real-time.
 80. The imagingsystem of claim 78, wherein the rolling shutter sensor is acomplementary metal-oxide-semiconductor (CMOS) imager.
 81. The imagingsystem of claim 78, wherein the rolling shutter sensor is integratedinto a chip-on-tip endoscope.
 82. The imaging system of claim 78,wherein the time-varying illumination source generates a stream ofequally spaced illumination pulses.
 83. The imaging system of claim 78,wherein a modulation frequency of the time-varying illumination sourceis significantly higher than a frame rate of the rolling shutter sensor.84. The imaging system of claim 78, wherein the time-varyingillumination source is a solid state illumination source.
 85. Theimaging system of claim 78, wherein the time-varying illumination sourceis a mode-locked laser that has a repetition rate synchronized with aline rate of the rolling shutter sensor.
 86. The imaging system of claim78, wherein the time-varying illumination source is a source ofnarrowband illumination.
 87. The imaging system of claim 78, wherein thetime-varying illumination source is a fluorescence illumination sourcewith a wavelength corresponding to an excitation spectrum of afluorescent agent in a scene.
 88. The imaging system of claim 78,wherein the time-varying illumination source is a structuredillumination source.
 89. The imaging system of claim 88, wherein theprocessor is further configured to perform depth estimation based on anillumination pattern output by the structured illumination source. 90.The imaging system of claim 78, wherein the processor is furtherconfigured to generate and transmit synchronized timing signals to eachof the rolling shutter sensor and the time-varying illumination source.